Task processing method, program, and system

ABSTRACT

A contribution value necessary for achieving a target accuracy from a correct answer probability of each participant is calculated, a contribution value of the participant is added for an answer in accordance with the calculation, and it is set at a condition for determining completion of a task, that is, determining that a correct answer is obtained and no additional participant is necessary. The contribution value is calculated as the inverse of the number of participants at which the target accuracy is reached with a predetermined correct answer probability. The contribution value of the participant is added to the contribution value for the task in which that participant participates. At the time when the sum of the contribution values for a task exceeds one or when one option is certain to be a correct answer, the result having the largest sum of the contribution values is output.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority under 35 U.S.C. § 119 from JapanesePatent Application No. 2012-220766 filed Oct. 2, 2012, the entirecontents of which are incorporated herein by reference.

BACKGROUND OF THE INVENTION

The present invention relates to a technique such as crowdsourcing forprocessing a task by people using a computer system, and morespecifically, it relates to a technique for ensuring accuracy of a taskoperation.

In recent years, as computer networks and web services have beendeveloped, a business form called crowdsourcing, which is outsourced toa large number of unspecified individuals, has increased.

Crowdsourcing is a technique for performing a task in a collectiveintelligence way by distributing a task, such as addition of captions toimages, transcribing speech information, or translation, that istechnically difficult to be automated by a computer and that can besubdivided, to many people as micro-tasks.

The use of crowdsourcing enables a service that could not be achieved byautomation by a computer, such as “service that returns description ofobjects appearing on any sent photograph.” Thus crowdsourcing is used invarious fields.

A problem in that case is that because of a characteristic ofparticipation of many workers vary in quality and the quality cannot beensured. For example, with the existing techniques, it is difficult toset a target of accuracy at 99%.

To ensure operation accuracy in crowdsourcing, techniques forintegrating results of a plurality of participants have been proposed.Some examples of such related techniques are described below.

One known example is a majority scheme that is a technique for making adecision by means of a majority decision of answers of participants.However, because the majority scheme cannot consider the skills ofparticipants, it has the drawback of low efficiency.

Another known example is a verification scheme using verification of anexpert. The verification scheme ensures accuracy in a sense thatverification based on expertise of an expert is conducted, but if afinite number of experts verify all tasks, the verifications cause abottleneck in processing.

Still another known example is an additional majority scheme. For thistechnique, two persons first process a task, and if results aredifferent, a majority decision is made. This technique has smallexpandability, and is required to combine with the verification scheme.To improve the above issues, techniques described in the papers listedbelow have been proposed.

P. Welinder, S. Branson, S. Belongie, and P. Perona, “Themultidimensional wisdom of crowds,” In Proceedings of the 24th AnnualConference on Neural Information Processing Systems (NIPS), 2010, pp.2424-2432 discloses improving the accuracy in majority decision byestimating the skills of workers in crowdsourcing and integratingopinions of the workers on the basis of the estimated skills. Thistechnique, however, cannot ensure target accuracy.

Pinar Donmez, Jaime G. Carbonell, and Jeff Schneider, “Efficientlylearning the accuracy of labeling sources for selective sampling,” InProceedings of the 15th ACM SIGKDD international conference on Knowledgediscovery and data mining (KDD '09), ACM, New York, N.Y., USA, 2009, pp.259-268 discloses finding reliable workers as fast as possible fromoperation histories of workers in crowdsourcing, causing only thereliable workers to perform a task, and making a majority decision. Thistechnique, however, is based on repeated operations by only reliableworkers. Thus it is difficult to deal with a situation where lessreliable workers are also contained to ensure a sufficient number ofworkers.

SUMMARY OF THE INVENTION

Accordingly, it is an object of the present invention to achievecrowdsourcing with a smaller number of persons at high operationaccuracy.

An aspect of the present invention is a method of processing a task on aserver computer using a joint activity of a plurality of participants atclient computers connected to the server computer, the server computerstoring at least one task Ti with N options wherein N is an integerequal to or larger than two, and a target accuracy for each task, themethod including the steps of receiving a result of an operation for thetask Ti from the client computer of each of the participants by theserver computer, calculating a contribution value of each of theparticipants from a correct answer probability at a stage where aplurality of results of operations for the task Ti are received throughthe client computers on the basis of the results of the operations bythe server computer, wherein the contribution value is an inverse of anumber of persons at which the participants each providing the correctanswer probability gather and the target accuracy is achieved, addingthe calculated contribution value of the participant to a contributionvalue for the task Ti performed by the participant by the servercomputer; and outputting a result occurring when the sum of thecontribution values for the task Ti exceeds a value that supports apredetermined condition as a result relating to the task Ti by theserver computer.

The present invention is conceived to solve the above problems, and isdirected to calculating a contribution value necessary for achieving atarget accuracy from a correct answer probability of each participant(worker), adding a contribution value of the participant for an answerin accordance with the calculation, and setting it at a condition fordetermining completion of a task, that is, determining that a correctanswer is obtained and no additional participant is necessary.

When a task is a majority decision task and a correct answer probabilityof a participant who participates in this majority decision task isconstant at P, the entire correct answer probability in the case where npersons participate in the majority decision at the correct answerprobability P is calculated by the following expression of a binominaldistribution.

$\begin{matrix}{{f\left( {P,n} \right)} = {\sum\limits_{m = 0}^{{\lceil\frac{n}{2}\rceil} - 1}{\begin{pmatrix}n \\{n - m}\end{pmatrix}\left( {\left( {1 - P} \right)^{m}P^{n - m}} \right)}}} & \left\lbrack {{Math}.\mspace{14mu} 1} \right\rbrack\end{matrix}$The sign┌ ┐  [Math. 2]above Σ in the above expression represents a ceiling function. Forexample,┌X┐  [Math. 3]indicates the smallest integer not less than X. The smallest number nwhen f(P,n) exceeds the target correct answer probability is consideredto be a minimally necessary number Q of persons on the basis of theabove expression, and 1/Q is set at a contribution value of theparticipant.

When the contribution value of the participant is calculated in theabove way, the contribution value of that participant is added to thecontribution value for the task in which the participant participates.The contribution value of that participant is also added to thecontribution value for the selected option for each answer. At the timewhen the sum of the contribution values for a task exceeds one or whenone option is certain to be a correct answer, a situation arises thateven if contribution values of remaining participants are added, thecontribution value for the option would not be exceeded by thecontribution value for each of the other options, the result having thelargest sum of the contribution values being the most likely is output.

The present invention may preferably implement the above-describedprocessing on a web server that receives answers from client computersof participants.

In the present invention, the following conditions may preferably beset.

-   -   N types of tasks exist. The tasks are different, but have the        same degree of difficulty.    -   A participant gives a correct answer at a probability higher        than random numbers or >0.5 in the case of two options.    -   The number of participants is finite.    -   (A subset of) any participants participate in a task. The order        is undetermined.    -   Participants perform a task only once.    -   A task having a “correct answer” and allowing a “correct answer        probability” to be calculated is a subject.

The present invention can provide an advantage of achieving a targetoperation accuracy using a smaller number of participants than that inthe related art by calculating a contribution value on the basis of abinomial distribution of correct answer probabilities of participants,adding the contribution value, and determining whether the targetaccuracy is achieved.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a whole hardware configuration for carrying out thepresent invention.

FIG. 2 is a block diagram of a hardware configuration of a clientcomputer.

FIG. 3 is a block diagram of a hardware configuration of a Web server.

FIG. 4 is a block diagram of a functional configuration according to anembodiment of the present invention.

FIG. 5 is a flowchart of processing according to the embodiment of thepresent invention.

FIG. 6 illustrates a screen used when a worker performs a task.

Embodiments of the present invention are described below with referenceto the drawings. These embodiments are the ones for use in describingpreferable forms of the present invention, and it is to be understoodthat they are not intended to limit the scope of the invention to thedescription here. The same reference numerals indicate the same objectsthroughout the drawings unless otherwise specified.

In FIG. 1, a plurality of client computers 106 a, 106 b, . . . , 106 zare connected to a Web server 102 over the Internet 104. In the systemillustrated in FIG. 1, a user of a client computer logs into the Webserver 102 over the lines of the Internet 104 through a Web browser.Specifically, the user inputs a predetermined uniform resource locator(URL) in the Web browser so that a predetermined page appears.

In login, a user of a client computer uses a given user ID and itsassociated password. When the user of the client computer logs in, atask with N options is given by the Web server 102. Here, N is aninteger equal to or larger than two. FIG. 6 illustrates an examplescreen appearing when a task with three options is displayed.

The Web server 102 is also connected to mobile terminals, such assmartphones 110 a, . . . , 110 k, over a packet communication network108. A user can also access an SNS in the Web server 102 through a Webbrowser installed in a smartphone using a given ID and its associatedpassword.

Next, the hardware block diagram of each of the client computersindicated with the reference numerals 106 a, 106 b, . . . , 106 z inFIG. 1 is described with reference to FIG. 2. In FIG. 2, the clientcomputer is typically a personal computer and includes a main memory206, a central processing unit (CPU) 204, and an integrated driveelectronics (IDE) controller 208. These components are connected to abus 202. The bus 202 is also connected to a display controller 214, acommunication interface 218, a universal serial bus (USB) interface 220,an audio interface 222, and a keyboard and mouse controller 228. The IDEcontroller 208 is connected to a hard disk drive (HDD) 210 and a digitalversatile disk (DVD) drive 212. The DVD drive 212 is used in installinga program from a compact-disk read-only memory (CD-ROM) or a DVD ifneeded. The display controller 214 may preferably be connected to adisplay device 216 having a liquid crystal display (LCD) screen. Thedisplay device 216 displays a screen for use in performing a taskthrough the Web browser. One example of the screen is illustrated inFIG. 6.

The USB interface 220 is used in connecting a device such as an externalHDD if needed.

The keyboard and mouse controller 228 is connected to a keyboard 230 anda mouse 232. The keyboard 230 and mouse 232 are used in logging in atask performing screen by a participant and selecting an answer from Noptions on a screen, such as the one illustrated in FIG. 6.

The CPU 204 may be any one based on a 32-bit architecture or a 64-bitarchitecture, for example. Examples of the CPU 204 can include Pentium(trademark of Intel Corporation) 4 and Core™ 2 Duo of Intel Corporationand Athlon™ of Advanced Micro Devices, Inc (AMD).

The hard disk drive 210 stores at least an operating system and a Webbrowser (not illustrated) running on the operating system. At the timeof the start-up of the system, the operating system is loaded into themain memory 206. Examples of the operating system can include Windows XP(trademark of Microsoft Corporation), Windows (trademark of MicrosoftCorporation) 7, Linux (trademark of Linus Torvalds), and Mac OS X.

The hard disk drive 210 further installs a Web browser. The Web browsermay preferably support JavaScript®, and examples of the Web browser caninclude, but not limited to, Internet Explorer, FireFox, and GoogleChrome.

The communication interface 218 communicates with the Web server 102using a TCP/IP communication function provided by the operating systemwith, for example, an Ethernet™ protocol.

The configuration of each of the client computers 106 a, 106 b, . . . ,106 z is described above. The smartphones 110 a, . . . , 110 k have aconfiguration substantially equivalent to that of each of the clientcomputers 106 a, 106 b, . . . , 106 z, and the description thereof isomitted. The Web browser in each of the smartphones 110 a, . . . , 110 kalso may preferably have the function of allowing JavaScript® to run.

FIG. 3 is a schematic block diagram of a hardware configuration of theWeb server 102. As illustrated in FIG. 3, the Internet 104, which isused for communicating with a client computer, and the packetcommunication network 108, which is used for communicating with asmartphone, are connected to a communication interface 302 of the Webserver 102. The communication interface 302 is further connected to abus 304. The bus 304 is connected to a CPU 306, a main memory (RAM) 308,and a hard disk drive (HDD) 310.

Although not illustrated, the Web server 102 may be further connected toa keyboard, a mouse, and a display, and these components may be used incontrolling the Web server 102 in general and performing a maintenanceoperation.

The hard disk drive 310 in the Web server 102 stores the operatingsystem and a correspondence table of user IDs and passwords for use inlog-in management for the client computers 106 a, 106 b, . . . , 106 z.The hard disk drive 310 further stores software such as Apache forenabling the Web server 102 to function as a Web server, and thesoftware is loaded into the main memory 308 and runs at the time of thestart-up of the Web server 102. With this, the client computers 106 a,106 b, . . . , 106 z can access the Web server 102 with a TCP/IPprotocol.

Although not illustrated, the hard disk drive 310 in the Web server 102also stores a communication module for communicating with a smartphoneover the packet communication network 108.

The hard disk drive 310 further stores a main program 402, accuracy data404, worker information 406, contribution value calculation module 408,a user response transmission and reception module 410, a task inputmodule 412, task data 414, a result display module 416, and a userinterface module 418. These components are described below withreference to the functional block diagram in FIG. 4.

As the above-described Web server 102, servers of models of IBM(trademark of International Business Machines Corporation) System X,System i, System p, and the like, which can be purchased fromInternational Business Machines Corporation, can be used. In that case,examples of a usable operating system can include AIX (trademark ofInternational Business Machines Corporation), UNIX (trademark of TheOpen Group), Linux (trademark of Linus Torvalds), and Windows™ 2003server. In particular, in the present embodiment, Linux (trademark ofLinus Torvalds) is used as the operating system, Java® EE is introducedthereon, server-side Java® is introduced, and the main program 402,contribution value calculation module 408, user response transmissionand reception module 410, task input module 412, result display module416, user interface module 418, and the like, which are processingprograms in the present invention may preferably be implemented as Java®programs running thereon or in the form of Servelet. They may also becreated using an existing programming language processing system, suchas C, C++, and C#.

The processing elements for carrying out the present invention aredescribed next with reference to the functional block diagram in FIG. 4.The main program 402 has the function of integrating the entireprocessing for achieving a predetermined operation accuracy andcompleting a task according to the present invention.

The accuracy data 404 is a preset target accuracy value for each oftasks T1, T2, . . . , Tn and is set in advance by an administrator oftask processing.

The worker information 406 is data that contains an ID and a correctanswer probability P of each registered worker, a login password of theworker, and other profile information.

The contribution value calculation module 408 has the function ofcalculating the smallest value n at which the following expressionexceeds a target accuracy when the correct answer probability P of aworker and the target accuracy for the task are given, considering it asthe number Q of persons necessary for achieving the target accuracy, andoutputting 1/Q as the contribution value.

$\begin{matrix}{{f\left( {P,n} \right)} = {\sum\limits_{m = 0}^{{\lceil\frac{n}{2}\rceil} - 1}{\begin{pmatrix}n \\{n - m}\end{pmatrix}\left( {\left( {1 - P} \right)^{m}P^{n - m}} \right)}}} & \left\lbrack {{Math}.\mspace{14mu} 4} \right\rbrack\end{matrix}$

The user response transmission and reception module 410 has the functionof receiving a set of data elements indicating a task ID, a user ID, anda result of selection from N options of the task from the worker throughthe user interface module 418 and of returning a response if needed.

The task input module 412 extracts data on a specified task from thetask data 414 in accordance with an instruction from the main program402 and communicates with client computers of a plurality of registeredworkers in unison through the user interface module 418. The task datacontains question items, the number of options, a message for eachoption, and the like to be displayed on the Web browser on each of theclient computers.

The result display module 416 displays an option that has the largestcontribution value of the options of the task as a true answer as aresult of processing on a screen of a client computer of, for example, auser who is logging in as an administrator.

Before processing in the present invention is described, premises of theprocessing are described next. The premises are as follows:

-   -   N types of tasks exist. The tasks are different, but have the        same degree of difficulty. That is, the target accuracies for        the tasks are the same.    -   Tasks having N options are assumed (N is an integer equal to or        larger than two). No task requiring a written answer is assumed.    -   A participant gives a correct answer at a probability higher        than random numbers (>0.5 in the case of two options).    -   The number of participants is finite.    -   (A subset of) any participants participate in a task. The order        is undetermined.    -   Participants perform a task only once.    -   A task having a “correct answer” and allowing a “correct answer        probability” to be calculated is a subject.    -   The average correct answer percentage of participants who        participated is used as the correct answer probability of a        participant who participates for the first time. The correct        answer probability of the participant who participates for the        first time may also be set in accordance with accomplishment        such as a career of some kind or a qualification.

In the premises described above, each of the degree of difficulty ofeach task and target accuracy therefor may not be set at the same valueand may be set at a different value, depending on the case.

The correct answer probability of a participant remains unchanged untilcompletion of processing after calculation of contribution value startsand the option having the largest contribution value is output. Acorrect answer probability of a participant who participated a pluralityof times is set in accordance with his or her past correct answerpercentage. This enables a participant who got a high correct answerpercentage to provide a high contribution value.

Here, referring to the flowchart in FIG. 5, users (participants) 420_1,420_2, . . . , 420_m attempt to log in the system illustrated in FIG. 4with their respective user IDs and predetermined passwords. The users420_1, 420_2, . . . , 420_m may log in from smartphones or may log infrom client computers, as illustrated in FIG. 1. The user IDs andpasswords sent from the users are sent to the main program 402 throughthe user interface module 418 and user response transmission andreception module 410. In response to this, the main program 402 refersto the worker information 406 and performs user authentication.

At step 502, the main program 402 calls the task input module 412,obtains data on a task Ti from the task data 414, and sends the data tothe users 420_1, 420_2, . . . , 420_m, who are logging in the task,through the user interface module 418. The main program 402 prepares avariable ti_c and a variable array ti_s[N] corresponding to the task Tiand initializes them to ti_c=0 and ti_s[0]=ti_s[1]= . . . =ti_s[N]=0.Here, ti_c is a variable that stores the entire contribution value ofthe task Ti, and ti_s[N] is a variable that stores a contribution valuefor each option.

The main program 402 further obtains data Ai on a predetermined accuracyfor the task Ti from the accuracy data 404.

In response to data transmission from the task input module 412, ascreen, such as the one displaying an explanation and radio buttonscorresponding to N options which are three options in this case,illustrated in FIG. 6, appears in the browser of the client computer ofeach of the users 420_1, 420_2, . . . , 420_m on the basis of thecontent stored as the task data 414 relating to the task Ti.

At step 504, in the case of the example illustrated in FIG. 6, a userwho has not yet given an answer provides an answer by reading theexplanation, clicking on the radio button corresponding to A, B, or Caccording to the instruction thereof, and clicking on OK. The answer issent to the main program 402 through the user interface module 418 anduser response transmission and reception module 410.

At step 506, the main program 402 obtains the correct answer probabilityP of the user from the worker information 406 on the basis of the userID corresponding to the received answer. Then the main program 402 callsthe contribution value calculation module 408 using the accuracy data Aiand correct answer probability P as arguments and calculates thecontribution value.

The contribution value calculation module 408 calculates n at whichf(P,n) just exceeds Ai using the function f(P,n) defined by thefollowing expression.

$\begin{matrix}{{f\left( {P,n} \right)} = {\sum\limits_{m = 0}^{{\lceil\frac{n}{2}\rceil} - 1}{\begin{pmatrix}n \\{n - m}\end{pmatrix}\left( {\left( {1 - P} \right)^{m}P^{n - m}} \right)}}} & \left\lbrack {{Math}.\mspace{14mu} 5} \right\rbrack\end{matrix}$

This can be expressed as calculation written in the form of pseudocodelike the C language as follows:

n=1;

while (f(P,n)<Ai)n⁺⁺;

f(P,n) is a monotone increasing function that approaches one as nincreases, 0<Ai<1, and thus the value of n can be easily found.

Using n obtained as a result of the calculation, 1/n is calculated asthe contribution value.

ti_c+=1/n; // The contribution value of the worker is added to thecontribution value for the task Ti.

ti_s[S]+=1/n; // The contribution value of the worker is added to thecontribution value for the option.

Here, S is the number of an option selected by the user, and S=kindicates that the (k+1)-th option is selected. In the case of theexample illustrated in FIG. 6, S=2.

At step 508, the main program 402 determines whether ti_c, which is thesum of the contribution values for Ti, is larger than the targetaccuracy (e.g., one) or whether any one of the options of Ti is certainto be a correct answer.

Here, the situation that any one of the options of Ti is certain to be acorrect answer is wherein the sum of the contribution values for themost likely option—the sum of the contribution values for the secondmost likely option>1—the sum of the contribution values of all of theparticipants performing the task is satisfied.

At step 508, when the main program 402 determines that the sum of thecontribution values for Ti does not exceed the target accuracy and thatnone of the options of Ti is certain to be a correct answer, theprocessing returns to step 504 and waits for an answer from a user whohas not yet given an answer.

In contrast, when ti_c, which is the sum of the contribution values forTi, is larger than the target accuracy or when any one of the options ofTi is certain to be a correct answer, at step 510, the main program 402determines that the option having the largest contribution value, thatis, the option j at which ti_s[j] is the largest is a true answer andinforms the administrator for the task of the result using the resultdisplay module 416.

In the processing about the task Ti described above, the main program402 may move to inputting of the next task at the time when the optionhaving the largest contribution value for one task is determined or mayinput different tasks to users at random.

The theoretical contribution values and the numbers of voters necessaryfor correct answer percentage 99% for predetermined correct answerprobabilities are provided in the table below.

TABLE 1 Number of Voters Correct Answer Necessary for CorrectTheoretical Probability Answer Percentage 99% Contribution Value 0.5113527 7.39262E−05 0.55 538 0.001858736 0.6 133 0.007518797 0.7 310.032258065 0.8 13 0.076923077 0.9 5 0.2 0.944 3 0.333333333 0.99 1 1

The embodiment is described above using an example established as aJava®-based client server system. The present invention is not limitedto a particular platform and can be achieved under any hardware,operating system, programming processing system or network environment.

Any task having N options can be used. The technique in the presentinvention is also applicable to a task in which a plurality of optionscan be selected at a time.

The invention claimed is:
 1. A method of processing a task on a servercomputer using a joint activity of a plurality of participants at clientcomputers connected to the server computer, the method comprising thesteps of: storing, on the server computer, at least one task Ti with Noptions, wherein N is an integer equal to or larger than two, and atarget accuracy for each task; receiving a result of an operation forthe task Ti from the client computer of each of the plurality ofparticipants by the server computer; calculating a contribution value ofeach of the plurality of participants from a correct answer probabilityat a stage where a plurality of results of operations for the task Tiare received through the client computers on the basis of the results ofthe operations by the server computer, wherein the contribution value isan inverse of a number of persons at which the plurality of participantseach providing the correct answer probability gather and the targetaccuracy is achieved, and wherein the contribution value is given bycalculating an entire correct answer probability when n personsparticipate in majority decision with correct answer probability P fromthe following expression:${{f\left( {P,n} \right)} = {\sum\limits_{m = 0}^{{\lceil\frac{n}{2}\rceil} - 1}{\begin{pmatrix}n \\{n - m}\end{pmatrix}\left( {\left( {1 - P} \right)^{m}P^{n - m}} \right)}}},$calculating a number Q of persons minimally necessary for achieving atarget correct answer probability on the basis of the above expression,and calculating a value of 1/Q as the contribution value; adding thecalculated contribution value of the participant to a contribution valuefor the task Ti performed by the participant by the server computer; andoutputting a result occurring when the sum of the contribution valuesfor the task Ti exceeds a value that supports a predetermined conditionas a result relating to the task Ti by the server computer.
 2. Themethod according to claim 1, wherein the predetermined condition is thatthe sum of the contribution values reaches one.
 3. The method accordingto claim 1, further comprising: a step of adding the contribution valuefor each option of the task, wherein the predetermined condition is thatone option is certain to be a correct answer.
 4. The method according toclaim 3, wherein being certain to be a correct answer is wherein the sumof the contribution values for the most likely option—the sum of thecontribution values for the second most likely option>1—the sum of thecontribution values of all of the participants performing the task issatisfied.
 5. The method according to claim 4, wherein the step ofoutputting the result as the result relating to the task Ti includes astep of outputting an option having the largest contribution value.
 6. Aprogram product for processing a task on a server computer using a jointactivity of a plurality of participants at client computers connected tothe server computer, the program product comprising a computer readablestorage medium having program instructions embodied therewith, whereinthe computer readable storage medium is not a transitory signal per se,the program instructions readable by the server computer to cause theserver computer to perform a method comprising: storing, on the servercomputer, at least one task Ti with N options, wherein N is an integerequal to or larger than two, and a target accuracy for each task;receiving a result of an operation for the task Ti from the clientcomputer of each of the plurality of participants by the servercomputer; calculating a contribution value of each of the plurality ofparticipants from a correct answer probability at a stage where aplurality of results of operations for the task Ti are received throughthe client computers on the basis of the results of the operations bythe server computer, the contribution value being an inverse of a numberof persons at which the plurality of participants each providing thecorrect answer probability gather and the target accuracy is achieved,and wherein the contribution value is given by calculating an entirecorrect answer probability when n persons participate in majoritydecision with correct answer probability P from the followingexpression:${{f\left( {P,n} \right)} = {\sum\limits_{m = 0}^{{\lceil\frac{n}{2}\rceil} - 1}{\begin{pmatrix}n \\{n - m}\end{pmatrix}\left( {\left( {1 - P} \right)^{m}P^{n - m}} \right)}}},$calculating a number Q of persons minimally necessary for achieving atarget correct answer probability on the basis of the above expression,and calculating a value of 1/Q as the contribution value; adding thecalculated contribution value of the participant to a contribution valuefor the task Ti performed by the participant by the server computer; andoutputting a result occurring when the sum of the contribution valuesfor the task Ti exceeds a value that supports a predetermined conditionas a result relating to the task Ti by the server computer.
 7. Theprogram product according to claim 6, wherein the predeterminedcondition is that the sum of the contribution values reaches one.
 8. Theprogram product according to claim 6, further comprising: a step ofadding the contribution value for each option of the task, wherein thepredetermined condition is that one option is certain to be a correctanswer.
 9. The program product according to claim 8, wherein beingcertain to be a correct answer is wherein the sum of the contributionvalues for the most likely option—the sum of the contribution values forthe second most likely option>1—the sum of the contribution values ofall of the participants performing the task, is satisfied.
 10. Theprogram product according to claim 9, wherein the step of outputting theresult as the result relating to the task Ti includes a step ofoutputting an option having the largest contribution value.
 11. A systemfor processing a task on a server computer using a joint activity of aplurality of participants at client computers connected to the servercomputer, the system comprising: a memory; and a processorcommunicatively coupled to the memory, wherein the processor isconfigured to: store, on the server computer, at least one task Ti withN options, wherein N is an integer equal to or larger than two, and atarget accuracy for each task; receive a result of an operation for thetask Ti from the client computer of each of the plurality ofparticipants; calculate a contribution value of each of the plurality ofparticipants from a correct answer probability at a stage where aplurality of results of operations for the task Ti are received throughthe client computers on the basis of the results of the operations, thecontribution value being an inverse of a number of persons at which theplurality of participants each providing the correct answer probabilitygather and the target accuracy is achieved, and wherein the contributionvalue is given by calculating an entire correct answer probability whenn persons participate in majority decision with correct answerprobability P from the following expression: $\begin{matrix}{{{f\left( {P,n} \right)} = {\sum\limits_{m = 0}^{{\lceil\frac{n}{2}\rceil} - 1}{\begin{pmatrix}n \\{n - m}\end{pmatrix}\left( {\left( {1 - P} \right)^{m}P^{n - m}} \right)}}},} & \;\end{matrix}$ calculating a number Q of persons minimally necessary forachieving a target correct answer probability on the basis of the aboveexpression, and calculating a value of 1/Q as the contribution value;add the calculated contribution value of the participant to acontribution value for the task Ti performed by the participant; andoutput a result occurring when the sum of the contribution values forthe task Ti exceeds a value that supports a predetermined condition as aresult relating to the task Ti.
 12. The system according to claim 11,wherein the predetermined condition is that the sum of the contributionvalues reaches one.
 13. The system according to claim 11, furthercomprising: the processor configured to: add the contribution value foreach option of the task, wherein the predetermined condition is that oneoption is certain to be a correct answer.
 14. The system according toclaim 13, wherein being certain to be a correct answer is wherein thesum of the contribution values for the most likely option—the sum of thecontribution values for the second most likely option>1—the sum of thecontribution values of all of the participants performing the task, issatisfied.
 15. The system according to claim 14, wherein the means foroutputting the result as the result relating to the task Ti outputs anoption having the largest contribution value.